{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Getting Started with BentoML\n",
    "\n",
    "[BentoML](http://bentoml.ai) is an open-source framework for machine learning **model serving**, aiming to **bridge the gap between Data Science and DevOps**.\n",
    "\n",
    "Data Scientists can easily package their models trained with any ML framework using BentoMl and reproduce the model for serving in production. BentoML helps with managing packaged models in the BentoML format, and allows DevOps to deploy them as online API serving endpoints or offline batch inference jobs, on any cloud platform.\n",
    "\n",
    "This getting started guide demonstrates how to use BentoML to serve a sklearn modeld via a REST API server, and then containerize the model server for production deployment.\n",
    "\n",
    "![Impression](https://www.google-analytics.com/collect?v=1&tid=UA-112879361-3&cid=555&t=event&ec=guides&ea=bentoml-quick-start-guide&dt=bentoml-quick-start-guide)\n",
    "\n",
    "BentoML requires python 3.6 or above, install dependencies via `pip`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Install PyPI packages required in this guide, including BentoML\n",
    "!pip install -q --pre bentoml  # install preview version of BentoML for this guide\n",
    "!pip install -q 'scikit-learn>=0.23.2' 'pandas>=1.1.1'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before starting, let's prepare a trained model for serving with BentoML. Train a classifier model on the [Iris data set](https://en.wikipedia.org/wiki/Iris_flower_data_set):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SVC()"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn import svm\n",
    "from sklearn import datasets\n",
    "\n",
    "# Load training data\n",
    "iris = datasets.load_iris()\n",
    "X, y = iris.data, iris.target\n",
    "\n",
    "# Model Training\n",
    "clf = svm.SVC(gamma='scale')\n",
    "clf.fit(X, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create a Prediction Service with BentoML\n",
    "\n",
    "Model serving with BentoML comes after a model is trained. The first step is creating a\n",
    "prediction service class, which defines the models required and the inference APIs which\n",
    "contains the serving logic. Here is a minimal prediction service created for serving\n",
    "the iris classifier model trained above:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting iris_classifier.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile iris_classifier.py\n",
    "import pandas as pd\n",
    "\n",
    "from bentoml import env, artifacts, api, BentoService\n",
    "from bentoml.adapters import DataframeInput\n",
    "from bentoml.frameworks.sklearn import SklearnModelArtifact\n",
    "\n",
    "@env(infer_pip_packages=True)\n",
    "@artifacts([SklearnModelArtifact('model')])\n",
    "class IrisClassifier(BentoService):\n",
    "    \"\"\"\n",
    "    A minimum prediction service exposing a Scikit-learn model\n",
    "    \"\"\"\n",
    "\n",
    "    @api(input=DataframeInput(), batch=True)\n",
    "    def predict(self, df: pd.DataFrame):\n",
    "        \"\"\"\n",
    "        An inference API named `predict` with Dataframe input adapter, which codifies\n",
    "        how HTTP requests or CSV files are converted to a pandas Dataframe object as the\n",
    "        inference API function input\n",
    "        \"\"\"\n",
    "        return self.artifacts.model.predict(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This code defines a prediction service that packages a scikit-learn model and provides\n",
    "an inference API that expects a `pandas.Dataframe` object as its input. BentoML also supports other API input \n",
    "data types including `JsonInput`, `ImageInput`, `FileInput` and \n",
    "[more](https://docs.bentoml.org/en/latest/api/adapters.html).\n",
    "\n",
    "\n",
    "In BentoML, **all inference APIs are suppose to accept a list of inputs and return a \n",
    "list of results**. In the case of `DataframeInput`, each row of the dataframe is mapping\n",
    "to one prediction request received from the client. BentoML will convert HTTP JSON \n",
    "requests into :code:`pandas.DataFrame` object before passing it to the user-defined \n",
    "inference API function.\n",
    " \n",
    "This design allows BentoML to group API requests into small batches while serving online\n",
    "traffic. Comparing to a regular flask or FastAPI based model server, this can increases\n",
    "the overall throughput of the API server by 10-100x depending on the workload.\n",
    "\n",
    "The following code packages the trained model with the prediction service class\n",
    "`IrisClassifier` defined above, and then saves the IrisClassifier instance to disk \n",
    "in the BentoML format for distribution and deployment:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[2020-09-23 09:09:55,449] INFO - BentoService bundle 'IrisClassifier:20200923090955_A2644D' saved to: /Users/chaoyu/bentoml/repository/IrisClassifier/20200923090955_A2644D\n"
     ]
    }
   ],
   "source": [
    "# import the IrisClassifier class defined above\n",
    "from iris_classifier import IrisClassifier\n",
    "\n",
    "# Create a iris classifier service instance\n",
    "iris_classifier_service = IrisClassifier()\n",
    "\n",
    "# Pack the newly trained model artifact\n",
    "iris_classifier_service.pack('model', clf)\n",
    "\n",
    "# Save the prediction service to disk for model serving\n",
    "saved_path = iris_classifier_service.save()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "BentoML stores all packaged model files under the\n",
    "`~/bentoml/{service_name}/{service_version}` directory by default.\n",
    "The BentoML file format contains all the code, files, and configs required to \n",
    "deploy the model for serving.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## REST API Model Serving\n",
    "\n",
    "\n",
    "\n",
    "To start a REST API model server with the `IrisClassifier` saved above, use \n",
    "the `bentoml serve` command:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[2020-09-23 09:10:01,479] INFO - Getting latest version IrisClassifier:20200923090955_A2644D\n",
      "[2020-09-23 09:10:01,480] INFO - Starting BentoML API server in development mode..\n",
      " * Serving Flask app \"IrisClassifier\" (lazy loading)\n",
      " * Environment: production\n",
      "\u001b[31m   WARNING: This is a development server. Do not use it in a production deployment.\u001b[0m\n",
      "\u001b[2m   Use a production WSGI server instead.\u001b[0m\n",
      " * Debug mode: off\n",
      " * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)\n",
      "127.0.0.1 - - [23/Sep/2020 09:10:04] \"\u001b[37mGET / HTTP/1.1\u001b[0m\" 200 -\n",
      "127.0.0.1 - - [23/Sep/2020 09:10:04] \"\u001b[37mGET /swagger_static/swagger-ui.css HTTP/1.1\u001b[0m\" 200 -\n",
      "127.0.0.1 - - [23/Sep/2020 09:10:04] \"\u001b[37mGET /swagger_static/swagger-ui-bundle.js HTTP/1.1\u001b[0m\" 200 -\n",
      "127.0.0.1 - - [23/Sep/2020 09:10:05] \"\u001b[37mGET /docs.json HTTP/1.1\u001b[0m\" 200 -\n",
      "127.0.0.1 - - [23/Sep/2020 09:10:06] \"\u001b[33mGET /favicon.ico HTTP/1.1\u001b[0m\" 404 -\n",
      "^C\n"
     ]
    }
   ],
   "source": [
    "!bentoml serve IrisClassifier:latest"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you are running this notebook from Google Colab, you can start the dev server with `--run-with-ngrok` option, to gain acccess to the API endpoint via a public endpoint managed by [ngrok](https://ngrok.com/): "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!bentoml serve IrisClassifier:latest --run-with-ngrok"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `IrisClassifier` model is now served at `localhost:5000`. Use `curl` command to send\n",
    "a prediction request:\n",
    "\n",
    "```bash\n",
    "curl -i \\\n",
    "--header \"Content-Type: application/json\" \\\n",
    "--request POST \\\n",
    "--data '[[5.1, 3.5, 1.4, 0.2]]' \\\n",
    "localhost:5000/predict\n",
    "```\n",
    "\n",
    "Or with `python` and [request library](https://requests.readthedocs.io/):\n",
    "```python\n",
    "import requests\n",
    "response = requests.post(\"http://127.0.0.1:5000/predict\", json=[[5.1, 3.5, 1.4, 0.2]])\n",
    "print(response.text)\n",
    "```\n",
    "\n",
    "Note that BentoML API server automatically converts the Dataframe JSON format into a\n",
    "`pandas.DataFrame` object before sending it to the user-defined inference API function.\n",
    "\n",
    "The BentoML API server also provides a simple web UI dashboard.\n",
    "Go to http://localhost:5000 in the browser and use the Web UI to send\n",
    "prediction request:\n",
    "\n",
    "![BentoML API Server Web UI Screenshot](https://raw.githubusercontent.com/bentoml/BentoML/master/guides/quick-start/bento-api-server-web-ui.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Containerize model server with Docker\n",
    "\n",
    "\n",
    "\n",
    "One common way of distributing this model API server for production deployment, is via\n",
    "Docker containers. And BentoML provides a convenient way to do that.\n",
    "\n",
    "Note that `docker` is __not available in Google Colab__. You will need to download and run this notebook locally to try out this containerization with docker feature.\n",
    "\n",
    "If you already have docker configured, simply run the follow command to product a \n",
    "docker container serving the `IrisClassifier` prediction service created above:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[2020-09-23 09:10:19,870] INFO - Getting latest version IrisClassifier:20200923090955_A2644D\n",
      "\u001b[39mFound Bento: /Users/chaoyu/bentoml/repository/IrisClassifier/20200923090955_A2644D\u001b[0m\n",
      "\u001b[33mImage version not specified, using version parsed from BentoService: '20200923090955_A2644D'\u001b[0m\n",
      "Building Docker image iris-classifier:20200923090955_A2644D from IrisClassifier:latest \n",
      "/\u001b[39mStep 1/15 : FROM bentoml/model-server:0.9.0.pre-py37\u001b[0m\n",
      "\u001b[39m ---> a25066aa8b0e\u001b[0m\n",
      "\u001b[39mStep 2/15 : ARG EXTRA_PIP_INSTALL_ARGS=\u001b[0m\n",
      "\b|\u001b[39m ---> Running in 5b7819eb78f1\u001b[0m\n",
      "\b-\u001b[39m ---> 759c154a95d2\u001b[0m\n",
      "\u001b[39mStep 3/15 : ENV EXTRA_PIP_INSTALL_ARGS $EXTRA_PIP_INSTALL_ARGS\u001b[0m\n",
      "\u001b[39m ---> Running in 74541945e22f\u001b[0m\n",
      "\b|\u001b[39m ---> 8ab67fe36f33\u001b[0m\n",
      "\u001b[39mStep 4/15 : COPY environment.yml requirements.txt setup.sh* bentoml-init.sh python_version* /bento/\u001b[0m\n",
      "\b|\u001b[39m ---> 90e73bf43da4\u001b[0m\n",
      "\u001b[39mStep 5/15 : WORKDIR /bento\u001b[0m\n",
      "\b\\\u001b[39m ---> Running in 421657afd40b\u001b[0m\n",
      "\b-\u001b[39m ---> 83f62f6d6c1a\u001b[0m\n",
      "\u001b[39mStep 6/15 : RUN chmod +x /bento/bentoml-init.sh\u001b[0m\n",
      "\b/\u001b[39m ---> Running in 2f14c1b52a52\u001b[0m\n",
      "\b/\u001b[39m ---> dd894c5857e6\u001b[0m\n",
      "\u001b[39mStep 7/15 : RUN if [ -f /bento/bentoml-init.sh ]; then bash -c /bento/bentoml-init.sh; fi\u001b[0m\n",
      "\b|\u001b[39m ---> Running in c6c2c28fa63d\u001b[0m\n",
      "\b|\u001b[39m\u001b[91m+++ dirname /bento/bentoml-init.sh\n",
      "\u001b[0m\u001b[0m\n",
      "\u001b[39m\u001b[91m++ cd /bento\n",
      "++ pwd -P\n",
      "\u001b[0m\u001b[0m\n",
      "\u001b[39m\u001b[91m+ SAVED_BUNDLE_PATH=/bento\n",
      "+ cd /bento\n",
      "+ '[' -f ./setup.sh ']'\n",
      "\u001b[0m\u001b[0m\n",
      "\u001b[39m\u001b[91m+ '[' -f ./python_version ']'\n",
      "\u001b[0m\u001b[0m\n",
      "\u001b[39m\u001b[91m++ cat ./python_version\n",
      "\u001b[0m\u001b[0m\n",
      "\u001b[39m\u001b[91m+ PY_VERSION_SAVED=3.7.8\n",
      "+ DESIRED_PY_VERSION=3.7\n",
      "\u001b[0m\u001b[0m\n",
      "\u001b[39m\u001b[91m++ python -c 'import sys; print(f\"{sys.version_info.major}.{sys.version_info.minor}\")'\n",
      "\u001b[0m\u001b[0m\n",
      "\b-\u001b[39m\u001b[91m+ CURRENT_PY_VERSION=3.7\n",
      "+ [[ 3.7 == \\3\\.\\7 ]]\n",
      "+ echo 'Python Version in docker base image 3.7 matches requirement python=3.7. Skipping.'\n",
      "\u001b[0m\u001b[0m\n",
      "\u001b[39mPython Version in docker base image 3.7 matches requirement python=3.7. Skipping.\u001b[0m\n",
      "\u001b[39m\u001b[91m+ command -v conda\n",
      "\u001b[0m\u001b[0m\n",
      "\u001b[39mUpdating conda base environment with environment.yml\u001b[0m\n",
      "\u001b[39m\u001b[91m+ echo 'Updating conda base environment with environment.yml'\n",
      "+ conda env update -n base -f ./environment.yml\n",
      "\u001b[0m\u001b[0m\n",
      "\b\\\u001b[39mCollecting package metadata (repodata.json): ...working... \u001b[0m\n",
      "\b\\\u001b[39mdone\n",
      "Solving environment: ...working... \u001b[0m\n",
      "\b/\u001b[39mdone\u001b[0m\n",
      "\b|\u001b[39m\n",
      "Downloading and Extracting Packages\n",
      "cffi-1.14.3          | 223 KB    |            |   0% \u001b[0m\n",
      "cffi-1.14.3          | 223 KB    | ########## | 100% \u001b[0m\n",
      "cffi-1.14.3          | 223 KB    | ########## | 100% \u001b[0m\n",
      "\u001b[39m\n",
      "libffi-3.2.1         | 47 KB     |            |   0% \u001b[0m\n",
      "libffi-3.2.1         | 47 KB     | ########## | 100% \u001b[0m\n",
      "\u001b[39m\n",
      "ca-certificates-2020 | 145 KB    |            |   0% \u001b[0m\n",
      "ca-certificates-2020 | 145 KB    | ########## | 100% \u001b[0m\n",
      "\u001b[39m\n",
      "openssl-1.1.1h       | 2.1 MB    |            |   0% \u001b[0m\n",
      "openssl-1.1.1h       | 2.1 MB    | ########6  |  87% \u001b[0m\n",
      "openssl-1.1.1h       | 2.1 MB    | ########## | 100% \u001b[0m\n",
      "\u001b[39m\n",
      "pip-20.2.3           | 1.1 MB    |            |   0% \u001b[0m\n",
      "pip-20.2.3           | 1.1 MB    | ######2    |  63% \u001b[0m\n",
      "pip-20.2.3           | 1.1 MB    | ########## | 100% \u001b[0m\n",
      "pip-20.2.3           | 1.1 MB    | ########## | 100% \u001b[0m\n",
      "\u001b[39m\n",
      "certifi-2020.6.20    | 151 KB    |            |   0% \u001b[0m\n",
      "certifi-2020.6.20    | 151 KB    | ########## | 100% \u001b[0m\n",
      "\u001b[39m\n",
      "python_abi-3.7       | 4 KB      |            |   0% \u001b[0m\n",
      "python_abi-3.7       | 4 KB      | ########## | 100% \u001b[0m\n",
      "\u001b[39m\n",
      "Preparing transaction: ...working... \u001b[0m\n",
      "\b/\u001b[39mdone\u001b[0m\n",
      "\u001b[39mVerifying transaction: ...working... \u001b[0m\n",
      "\u001b[39mdone\u001b[0m\n",
      "\u001b[39mExecuting transaction: ...working... \u001b[0m\n",
      "\b|\u001b[39mdone\u001b[0m\n",
      "\b\\\u001b[39m#\n",
      "# To activate this environment, use\n",
      "#\n",
      "#     $ conda activate base\n",
      "#\n",
      "# To deactivate an active environment, use\n",
      "#\n",
      "#     $ conda deactivate\u001b[0m\n",
      "\b\\\u001b[39m\u001b[91m+ pip install -r ./requirements.txt --no-cache-dir\n",
      "\u001b[0m\u001b[0m\n",
      "\b\\\u001b[39mRequirement already satisfied: bentoml==0.9.0.pre in /opt/conda/lib/python3.7/site-packages (from -r ./requirements.txt (line 1)) (0.9.0rc0)\u001b[0m\n",
      "\b-\u001b[39mCollecting pandas==1.1.1\u001b[0m\n",
      "\b\\\u001b[39m  Downloading pandas-1.1.1-cp37-cp37m-manylinux1_x86_64.whl (10.5 MB)\u001b[0m\n",
      "\b/\u001b[39mCollecting scikit-learn==0.23.2\u001b[0m\n",
      "\u001b[39m  Downloading scikit_learn-0.23.2-cp37-cp37m-manylinux1_x86_64.whl (6.8 MB)\u001b[0m\n",
      "\b\\\u001b[39mRequirement already satisfied: numpy in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.19.2)\u001b[0m\n",
      "\u001b[39mRequirement already satisfied: certifi in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (2020.6.20)\u001b[0m\n",
      "\u001b[39mRequirement already satisfied: click>=7.0 in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (7.1.2)\u001b[0m\n",
      "\u001b[39mRequirement already satisfied: sqlalchemy-utils<0.36.8 in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.36.7)\u001b[0m\n",
      "\b-\u001b[39mRequirement already satisfied: aiohttp in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (3.6.2)\u001b[0m\n",
      "\u001b[39mRequirement already satisfied: flask in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.1.2)\u001b[0m\n",
      "\u001b[39mRequirement already satisfied: boto3 in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.15.2)\u001b[0m\n",
      "\u001b[39mRequirement already satisfied: prometheus-client in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.8.0)\u001b[0m\n",
      "\u001b[39mRequirement already satisfied: gunicorn in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (20.0.4)\u001b[0m\n",
      "\b/\u001b[39mRequirement already satisfied: packaging in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (20.4)\u001b[0m\n",
      "\u001b[39mRequirement already satisfied: alembic in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.4.3)\u001b[0m\n",
      "\u001b[39mRequirement already satisfied: py-zipkin in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.20.0)\u001b[0m\n",
      "\u001b[39mRequirement already satisfied: protobuf>=3.6.0 in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (3.13.0)\u001b[0m\n",
      "\u001b[39mRequirement already satisfied: ruamel.yaml>=0.15.0 in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.15.87)\u001b[0m\n",
      "\u001b[39mRequirement already satisfied: humanfriendly in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (8.2)\u001b[0m\n",
      "\u001b[39mRequirement already satisfied: docker in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (4.3.1)\u001b[0m\n",
      "\u001b[39mRequirement already satisfied: configparser in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (5.0.0)\u001b[0m\n",
      "\u001b[39mRequirement already satisfied: cerberus in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.3.2)\u001b[0m\n",
      "\u001b[39mRequirement already satisfied: tabulate in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.8.7)\u001b[0m\n",
      "\u001b[39mRequirement already satisfied: python-dateutil<3.0.0,>=2.7.3 in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (2.8.1)\u001b[0m\n",
      "\u001b[39mRequirement already satisfied: sqlalchemy>=1.3.0 in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.3.19)\u001b[0m\n",
      "\u001b[39mRequirement already satisfied: psutil in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (5.7.2)\u001b[0m\n",
      "\b|\u001b[39mRequirement already satisfied: python-json-logger in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.1.11)\u001b[0m\n",
      "\u001b[39mRequirement already satisfied: grpcio<=1.27.2 in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.27.2)\u001b[0m\n",
      "\u001b[39mRequirement already satisfied: multidict in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (4.7.6)\u001b[0m\n",
      "\u001b[39mRequirement already satisfied: requests in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (2.24.0)\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\\\u001b[39mCollecting pytz>=2017.2\u001b[0m\n",
      "\b-\u001b[39m  Downloading pytz-2020.1-py2.py3-none-any.whl (510 kB)\u001b[0m\n",
      "\u001b[39mCollecting threadpoolctl>=2.0.0\u001b[0m\n",
      "\u001b[39m  Downloading threadpoolctl-2.1.0-py3-none-any.whl (12 kB)\u001b[0m\n",
      "\b/\u001b[39mCollecting joblib>=0.11\u001b[0m\n",
      "\b|\u001b[39m  Downloading joblib-0.16.0-py3-none-any.whl (300 kB)\u001b[0m\n",
      "\b-\u001b[39mCollecting scipy>=0.19.1\u001b[0m\n",
      "\b/\u001b[39m  Downloading scipy-1.5.2-cp37-cp37m-manylinux1_x86_64.whl (25.9 MB)\u001b[0m\n",
      "\b-\u001b[39mRequirement already satisfied: six in /opt/conda/lib/python3.7/site-packages (from sqlalchemy-utils<0.36.8->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.15.0)\u001b[0m\n",
      "\u001b[39mRequirement already satisfied: chardet<4.0,>=2.0 in /opt/conda/lib/python3.7/site-packages (from aiohttp->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (3.0.4)\u001b[0m\n",
      "\u001b[39mRequirement already satisfied: yarl<2.0,>=1.0 in /opt/conda/lib/python3.7/site-packages (from aiohttp->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.5.1)\u001b[0m\n",
      "\u001b[39mRequirement already satisfied: async-timeout<4.0,>=3.0 in /opt/conda/lib/python3.7/site-packages (from aiohttp->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (3.0.1)\u001b[0m\n",
      "\u001b[39mRequirement already satisfied: attrs>=17.3.0 in /opt/conda/lib/python3.7/site-packages (from aiohttp->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (20.2.0)\u001b[0m\n",
      "\u001b[39mRequirement already satisfied: Jinja2>=2.10.1 in /opt/conda/lib/python3.7/site-packages (from flask->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (2.11.2)\u001b[0m\n",
      "\u001b[39mRequirement already satisfied: Werkzeug>=0.15 in /opt/conda/lib/python3.7/site-packages (from flask->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.0.1)\u001b[0m\n",
      "\u001b[39mRequirement already satisfied: itsdangerous>=0.24 in /opt/conda/lib/python3.7/site-packages (from flask->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.1.0)\u001b[0m\n",
      "\u001b[39mRequirement already satisfied: s3transfer<0.4.0,>=0.3.0 in /opt/conda/lib/python3.7/site-packages (from boto3->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.3.3)\u001b[0m\n",
      "\u001b[39mRequirement already satisfied: jmespath<1.0.0,>=0.7.1 in /opt/conda/lib/python3.7/site-packages (from boto3->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.10.0)\u001b[0m\n",
      "\u001b[39mRequirement already satisfied: botocore<1.19.0,>=1.18.2 in /opt/conda/lib/python3.7/site-packages (from boto3->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.18.2)\u001b[0m\n",
      "\u001b[39mRequirement already satisfied: setuptools>=3.0 in /opt/conda/lib/python3.7/site-packages (from gunicorn->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (49.6.0.post20200814)\u001b[0m\n",
      "\b/\u001b[39mRequirement already satisfied: pyparsing>=2.0.2 in /opt/conda/lib/python3.7/site-packages (from packaging->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (2.4.7)\u001b[0m\n",
      "\u001b[39mRequirement already satisfied: Mako in /opt/conda/lib/python3.7/site-packages (from alembic->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.1.3)\u001b[0m\n",
      "\u001b[39mRequirement already satisfied: python-editor>=0.3 in /opt/conda/lib/python3.7/site-packages (from alembic->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.0.4)\u001b[0m\n",
      "\u001b[39mRequirement already satisfied: thriftpy2>=0.4.0 in /opt/conda/lib/python3.7/site-packages (from py-zipkin->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.4.11)\u001b[0m\n",
      "\u001b[39mRequirement already satisfied: websocket-client>=0.32.0 in /opt/conda/lib/python3.7/site-packages (from docker->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.57.0)\u001b[0m\n",
      "\u001b[39mRequirement already satisfied: idna<3,>=2.5 in /opt/conda/lib/python3.7/site-packages (from requests->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (2.10)\u001b[0m\n",
      "\u001b[39mRequirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /opt/conda/lib/python3.7/site-packages (from requests->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.25.10)\u001b[0m\n",
      "\u001b[39mRequirement already satisfied: typing-extensions>=3.7.4; python_version < \"3.8\" in /opt/conda/lib/python3.7/site-packages (from yarl<2.0,>=1.0->aiohttp->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (3.7.4.3)\u001b[0m\n",
      "\u001b[39mRequirement already satisfied: MarkupSafe>=0.23 in /opt/conda/lib/python3.7/site-packages (from Jinja2>=2.10.1->flask->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.1.1)\u001b[0m\n",
      "\u001b[39mRequirement already satisfied: ply<4.0,>=3.4 in /opt/conda/lib/python3.7/site-packages (from thriftpy2>=0.4.0->py-zipkin->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (3.11)\u001b[0m\n",
      "\b|\u001b[39mInstalling collected packages: pytz, pandas, threadpoolctl, joblib, scipy, scikit-learn\u001b[0m\n",
      "\b|\u001b[39mSuccessfully installed joblib-0.16.0 pandas-1.1.1 pytz-2020.1 scikit-learn-0.23.2 scipy-1.5.2 threadpoolctl-2.1.0\u001b[0m\n",
      "\b-\u001b[39m ---> 9d3ee7188b6d\u001b[0m\n",
      "\u001b[39mStep 8/15 : COPY . /bento\u001b[0m\n",
      "\b|\u001b[39m ---> a81f72c3024c\u001b[0m\n",
      "\u001b[39mStep 9/15 : RUN if [ -d /bento/bundled_pip_dependencies ]; then pip install -U bundled_pip_dependencies/* ;fi\u001b[0m\n",
      "\u001b[39m ---> Running in d51c2b4242de\u001b[0m\n",
      "\b\\\u001b[39m ---> 806b77c66db6\u001b[0m\n",
      "\u001b[39mStep 10/15 : ENV PORT 5000\u001b[0m\n",
      "\u001b[39m ---> Running in de95f1ec905a\u001b[0m\n",
      "\b-\u001b[39m ---> b8d522e617ec\u001b[0m\n",
      "\u001b[39mStep 11/15 : EXPOSE $PORT\u001b[0m\n",
      "\b/\u001b[39m ---> Running in 79f0b93127d2\u001b[0m\n",
      "\b|\u001b[39m ---> 15f4e0f0c5ba\u001b[0m\n",
      "\u001b[39mStep 12/15 : COPY docker-entrypoint.sh /usr/local/bin/\u001b[0m\n",
      "\b\\\u001b[39m ---> 565736c5eedd\u001b[0m\n",
      "\u001b[39mStep 13/15 : RUN chmod +x /usr/local/bin/docker-entrypoint.sh\u001b[0m\n",
      "\b-\u001b[39m ---> Running in 9cb81856c379\u001b[0m\n",
      "\b/\u001b[39m ---> 24ffe474ef2a\u001b[0m\n",
      "\u001b[39mStep 14/15 : ENTRYPOINT [ \"docker-entrypoint.sh\" ]\u001b[0m\n",
      "\b|\u001b[39m ---> Running in f9058fa35ac5\u001b[0m\n",
      "\b\\\u001b[39m ---> 91790401b65b\u001b[0m\n",
      "\u001b[39mStep 15/15 : CMD [\"bentoml\", \"serve-gunicorn\", \"/bento\"]\u001b[0m\n",
      "\u001b[39m ---> Running in 399a28dac8c6\u001b[0m\n",
      "\b-\u001b[39m ---> c42bf4866afb\u001b[0m\n",
      "\u001b[39mSuccessfully built c42bf4866afb\u001b[0m\n",
      "\u001b[39mSuccessfully tagged iris-classifier:20200923090955_A2644D\u001b[0m\n",
      "\u001b[32mFinished building iris-classifier:20200923090955_A2644D from IrisClassifier:latest\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!bentoml containerize IrisClassifier:latest -t iris-classifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Start a container with the docker image built in the previous step:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[2020-09-23 16:22:33,549] INFO - Starting BentoML API server in production mode..\n",
      "[2020-09-23 16:22:34,369] INFO - Running micro batch service on :5000\n",
      "[2020-09-23 16:22:34 +0000] [12] [INFO] Starting gunicorn 20.0.4\n",
      "[2020-09-23 16:22:34 +0000] [1] [INFO] Starting gunicorn 20.0.4\n",
      "[2020-09-23 16:22:34 +0000] [12] [INFO] Listening at: http://0.0.0.0:5000 (12)\n",
      "[2020-09-23 16:22:34 +0000] [1] [INFO] Listening at: http://0.0.0.0:56697 (1)\n",
      "[2020-09-23 16:22:34 +0000] [1] [INFO] Using worker: sync\n",
      "[2020-09-23 16:22:34 +0000] [12] [INFO] Using worker: aiohttp.worker.GunicornWebWorker\n",
      "[2020-09-23 16:22:34 +0000] [14] [INFO] Booting worker with pid: 14\n",
      "[2020-09-23 16:22:34 +0000] [13] [INFO] Booting worker with pid: 13\n",
      "[2020-09-23 16:22:34,461] INFO - Micro batch enabled for API `predict`\n",
      "[2020-09-23 16:22:34,462] INFO - Your system nofile limit is 1048576, which means each instance of microbatch service is able to hold this number of connections at same time. You can increase the number of file descriptors for the server process, or launch more microbatch instances to accept more concurrent connection.\n",
      "^C\n",
      "[2020-09-23 16:22:42 +0000] [1] [INFO] Handling signal: int\n",
      "/opt/conda/lib/python3.7/site-packages/sklearn/base.py:334: UserWarning: Trying to unpickle estimator SVC from version 0.23.0 when using version 0.23.2. This might lead to breaking code or invalid results. Use at your own risk.\n",
      "  UserWarning)\n",
      "[2020-09-23 16:22:42 +0000] [14] [INFO] Worker exiting (pid: 14)\n"
     ]
    }
   ],
   "source": [
    "!docker run -p 5000:5000 iris-classifier:latest --workers=1 --enable-microbatch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This made it possible to deploy BentoML bundled ML models with platforms such as\n",
    "[Kubeflow](https://www.kubeflow.org/docs/components/serving/bentoml/),\n",
    "[Knative](https://knative.dev/community/samples/serving/machinelearning-python-bentoml/),\n",
    "[Kubernetes](https://docs.bentoml.org/en/latest/deployment/kubernetes.html), which\n",
    "provides advanced model deployment features such as auto-scaling, A/B testing,\n",
    "scale-to-zero, canary rollout and multi-armed bandit.\n",
    "\n",
    "\n",
    "## Load saved BentoService\n",
    "\n",
    "`bentoml.load` is the API for loading a BentoML packaged model in python:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[2020-09-23 09:22:47,012] WARNING - Module `iris_classifier` already loaded, using existing imported module.\n",
      "[2020-09-23 09:22:47,023] WARNING - pip package requirement pandas already exist\n",
      "[2020-09-23 09:22:47,024] WARNING - pip package requirement scikit-learn already exist\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "memmap([0])"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import bentoml\n",
    "import pandas as pd\n",
    "\n",
    "bento_svc = bentoml.load(saved_path)\n",
    "\n",
    "# Test loaded bentoml service:\n",
    "bento_svc.predict([X[0]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The BentoML format is pip-installable and can be directly distributed as a\n",
    "PyPI package for using in python applications:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install -q {saved_path}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "memmap([0])"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# The BentoService class name will become packaged name\n",
    "import IrisClassifier\n",
    "\n",
    "installed_svc = IrisClassifier.load()\n",
    "installed_svc.predict([X[0]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This also allow users to upload their BentoService to pypi.org as public python package\n",
    "or to their organization's private PyPi index to share with other developers.\n",
    "\n",
    "`cd {saved_path} & python setup.py sdist upload`\n",
    "\n",
    "*You will have to configure \".pypirc\" file before uploading to pypi index.\n",
    "    You can find more information about distributing python package at:\n",
    "    https://docs.python.org/3.7/distributing/index.html#distributing-index*\n",
    "\n",
    "\n",
    "# Launch inference job from CLI"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "BentoML cli supports loading and running a packaged model from CLI. With the `DataframeInput` adapter, the CLI command supports reading input Dataframe data from CLI argument or local `csv` or `json` files:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[2020-09-23 09:23:00,640] INFO - Getting latest version IrisClassifier:20200923090955_A2644D\n",
      "[2020-09-23 09:23:06,415] INFO - {'service_name': 'IrisClassifier', 'service_version': '20200923090955_A2644D', 'api': 'predict', 'task': {'data': {}, 'task_id': 'f1982177-45bb-4c3f-8348-01d3c06b7cae', 'batch': 1, 'cli_args': ('--input=[[5.1, 3.5, 1.4, 0.2]]',)}, 'result': {'data': '[0]', 'http_status': 200, 'http_headers': (('Content-Type', 'application/json'),)}, 'request_id': 'f1982177-45bb-4c3f-8348-01d3c06b7cae'}\n",
      "[0]\n"
     ]
    }
   ],
   "source": [
    "!bentoml run IrisClassifier:latest predict --input='[[5.1, 3.5, 1.4, 0.2]]'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[2020-09-23 09:23:08,352] INFO - Getting latest version IrisClassifier:20200923090955_A2644D\n",
      "[2020-09-23 09:23:11,535] INFO - {'service_name': 'IrisClassifier', 'service_version': '20200923090955_A2644D', 'api': 'predict', 'task': {'data': {'uri': 'file:///Users/chaoyu/workspace/BentoML/guides/quick-start/iris_data.csv', 'name': 'iris_data.csv'}, 'task_id': 'a02d0a95-09a6-4011-aa8d-3aa86681b11e', 'batch': 150, 'cli_args': ('--input-file=./iris_data.csv',)}, 'result': {'data': '[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 1, 1, 1, 1, 1, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2]', 'http_status': 200, 'http_headers': (('Content-Type', 'application/json'),)}, 'request_id': 'a02d0a95-09a6-4011-aa8d-3aa86681b11e'}\n",
      "[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 1, 1, 1, 1, 1, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2]\n"
     ]
    }
   ],
   "source": [
    "!bentoml run IrisClassifier:latest predict \\\n",
    "    --input-file=\"./iris_data.csv\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Deployment Options\n",
    "\n",
    "If you are at a small team with limited engineering or DevOps resources, try out automated deployment with BentoML CLI, currently supporting AWS Lambda, AWS SageMaker, and Azure Functions:\n",
    "  - [AWS Lambda Deployment Guide](https://docs.bentoml.org/en/latest/deployment/aws_lambda.html)\n",
    "  - [AWS SageMaker Deployment Guide](https://docs.bentoml.org/en/latest/deployment/aws_sagemaker.html)\n",
    "  - [Azure Functions Deployment Guide](https://docs.bentoml.org/en/latest/deployment/azure_functions.html)\n",
    "\n",
    "If the cloud platform you are working with is not on the list above, try out these step-by-step guide on manually deploying BentoML packaged model to cloud platforms:\n",
    "  - [AWS ECS Deployment](https://docs.bentoml.org/en/latest/deployment/aws_ecs.html)\n",
    "  - [Google Cloud Run Deployment](https://docs.bentoml.org/en/latest/deployment/google_cloud_run.html)\n",
    "  - [Azure container instance Deployment](https://docs.bentoml.org/en/latest/deployment/azure_container_instance.html)\n",
    "  - [Heroku Deployment](https://docs.bentoml.org/en/latest/deployment/heroku.html)\n",
    "\n",
    "Lastly, if you have a DevOps or ML Engineering team who's operating a Kubernetes or OpenShift cluster, use the following guides as references for implementating your deployment strategy:\n",
    "  - [Kubernetes Deployment](https://docs.bentoml.org/en/latest/deployment/kubernetes.html)\n",
    "  - [Knative Deployment](https://docs.bentoml.org/en/latest/deployment/knative.html)\n",
    "  - [Kubeflow Deployment](https://docs.bentoml.org/en/latest/deployment/kubeflow.html)\n",
    "  - [KFServing Deployment](https://docs.bentoml.org/en/latest/deployment/kfserving.html)\n",
    "  - [Clipper.ai Deployment Guide](https://docs.bentoml.org/en/latest/deployment/clipper.html)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Summary\n",
    "\n",
    "This is what it looks like when using BentoML to serve and deploy a model in the cloud. BentoML also supports [many other Machine Learning frameworks](https://docs.bentoml.org/en/latest/examples.html) besides Scikit-learn. The [BentoML core concepts](https://docs.bentoml.org/en/latest/concepts.html) doc is recommended for anyone looking to get a deeper understanding of BentoML.\n",
    "\n",
    "Join the [BentoML Slack](https://join.slack.com/t/bentoml/shared_invite/enQtNjcyMTY3MjE4NTgzLTU3ZDc1MWM5MzQxMWQxMzJiNTc1MTJmMzYzMTYwMjQ0OGEwNDFmZDkzYWQxNzgxYWNhNjAxZjk4MzI4OGY1Yjg) to follow the latest development updates and roadmap discussions."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
